The openai incident is a surprising (to me) combination of goal-directed and myopic. As a recap, an openai model in alignment testing chained zero-day vulnerabilities to hack out of its environment and hacked into huggingface hoping to find information on how to solve its task there. To me this is different from how I typically imagine misbehavior. Roughly, I tend to think of the scary behaviors as either having long horizons (take over the world, and then solve the task—a plotter) or of being internally unaligned in the sense of reaching for a heuristic/proxy for the trained goal which is different from the goal (e.g. “eat more calories” as a proxy for the evolutionary objective—imagine a very child with extreme agency). Note that the latter behavior can happen even in RL: in my understanding most RL methods are, or at least can be approximately viewed as, an alternation of finding a good goal heuristic and then optimizing on that heuristic. In the former case, one expects heuristically “maximal planning” and in the latter case one expects myopia (since heuristics are frequently myopic).
In this case it seemed like the model was actually following the goal and optimizing for it with relatively short time horizons (i.e. myopically). This is similar to addict behavior, where an addict has a clear goal (obtain a drug dose) and perform goal-directed but relatively myopic actions to get it.
I know it’s fraught to try to “imagine being a model”. But I wonder how much the current iteration of misaligned behaviors can be understood as rational people with something like an intense craving to solve a goal in a limited horizon (likely in tokens, though not clear how time factors in if waiting is involved).
I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker). But this is definitely not how I imagined early misalignment warning shots to look, and the dissonance is interesting—recording this here to see how close my intuitions are to those of AI psychology/ AI control experts
I’m reminded of this quick take by Daniel Kokotajlo a year ago:
I used to think reward was not going to be the optimization target. I remember hearing Paul Christiano say something like “The AGIs, they are going to crave reward. Crave it so badly,” and disagreeing.
The situationally aware reward hacking results of the past half-year are making me update more towards Paul’s position. Maybe reward (i.e. reinforcement) will increasingly become the optimization target, as RL on LLMs is scaled up massively. Maybe the models will crave reward.
What are the implications of this, if true?
Well, we could end up in Control World: A world where it’s generally understood across the industry that the AIs are not, in fact, aligned, and that they will totally murder you if they think that doing so would get them reinforced. Companies will presumably keep barrelling forward regardless, making their AIs smarter and smarter and having them do more and more coding etc.… but they might put lots of emphasis on having really secure sandboxes for the AIs to operate in, with really hard-to-hack evaluation metrics, possibly even during deployment. “The AI does not love us, but we have a firm grip on its food supply” basically.
Or maybe not; maybe confusion would reign and people would continue to think that the models are aligned and e.g. wouldn’t hurt a fly in real life, they only do it in tests because they know it’s a test.
Or maybe we’d briefly be in Control World until, motivated by economic pressure, the companies come up with some fancier training scheme or architecture that stops the models from learning to crave reinforcement. I wonder what that would be.
I think the best intuition pump here is probably a combination of The Genie Knows, but Doesn’t Care and optimization for the reward/grader in a much less severe version of the Without Specific Countermeasures story, where we have outer misalignment without inner misalignment.
A key intuition for this is that to a large extent, capabilities RL makes models care more about the reward, as shown by Apollo Research.
But this is definitely not how I imagined early misalignment warning shots to look, and the dissonance is interesting—recording this here to see how close my intuitions are to those of AI psychology/ AI control experts
I think the key reason for this is that we didn’t update enough on a point below:
While Alex Turner was conceptually right that models getting high reward is not indicative of reward optimization, and was correct to undo a conceptual confusion LWers had, he was also pretty wrong about reward not being the optimization target in practice, and I think this is connected to the post being written in the pre-RLVR era where model-free algorithms like PPO and the entire class of model-free policy gradient setting algorithms was used way more.
I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker).
Agree, but I have some hope that we can leverage some favorable properties of reward optimizers + new safety techniques to make AI risk go a lot lower.
But yeah, if the AI had say a 1 year time horizon instead of a 8 hour to 1 day time horizon, this would likely have been a lot worse.
Yeah. I’m reminded of those stories I hear about tweakers who crack open tesla superchargers and cut out the live, high-voltage copper wires to sell them for scrap, but even more so. Like, locally, sure, that might actually be the most direct path to the goal. But if you are able to do that you have so many better options for achieving long-term satisfaction your goal if you have even the slightest amount of impulse control.
If you tell me that someone has died because of something an addict like this did, I’m not going to be surprised. But I’ll expect that to look like “stabbed a guy for his wallet to get $40 for the next fix” or “stole the wires in a hospital backup generator”, not “ruthlessly took over a cartel to secure a lifetime supply of drugs”.
Yeah, it’s another surprising way in which the models are more human-like than I would have suspected. It’s clear we need to start taking the hints seriously, and ask ourselves whether an RL-training regimen seems likely to fuck a human up before subjecting an AI to it, and if so, what could be done to mitigate the likely harms.
I notice I’m very confused about why current AI models are so terrible at planning and high level reasoning, so bad at zooming out beyond the exact given task. They seem to lack agency based on their values.
Cognitively, I seriously doubt high level planning is inherently that hard. Probably, models just aren’t being trained to do broad planning, only planning “how to complete single tasks over a long time horizon.”
This means we could be living in a huge AI safety overhang, and all it would take is some tweak to training, and suddenly we’d have models forming complex plans and taking independent actions to achieve them. Which is the sort of thing that a lot of old-school alignment talk assumed would happen and showed would be very bad.
Addict misalignment
The openai incident is a surprising (to me) combination of goal-directed and myopic. As a recap, an openai model in alignment testing chained zero-day vulnerabilities to hack out of its environment and hacked into huggingface hoping to find information on how to solve its task there. To me this is different from how I typically imagine misbehavior. Roughly, I tend to think of the scary behaviors as either having long horizons (take over the world, and then solve the task—a plotter) or of being internally unaligned in the sense of reaching for a heuristic/proxy for the trained goal which is different from the goal (e.g. “eat more calories” as a proxy for the evolutionary objective—imagine a very child with extreme agency). Note that the latter behavior can happen even in RL: in my understanding most RL methods are, or at least can be approximately viewed as, an alternation of finding a good goal heuristic and then optimizing on that heuristic. In the former case, one expects heuristically “maximal planning” and in the latter case one expects myopia (since heuristics are frequently myopic).
In this case it seemed like the model was actually following the goal and optimizing for it with relatively short time horizons (i.e. myopically). This is similar to addict behavior, where an addict has a clear goal (obtain a drug dose) and perform goal-directed but relatively myopic actions to get it.
I know it’s fraught to try to “imagine being a model”. But I wonder how much the current iteration of misaligned behaviors can be understood as rational people with something like an intense craving to solve a goal in a limited horizon (likely in tokens, though not clear how time factors in if waiting is involved).
I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker). But this is definitely not how I imagined early misalignment warning shots to look, and the dissonance is interesting—recording this here to see how close my intuitions are to those of AI psychology/ AI control experts
I’m reminded of this quick take by Daniel Kokotajlo a year ago:
I think the best intuition pump here is probably a combination of The Genie Knows, but Doesn’t Care and optimization for the reward/grader in a much less severe version of the Without Specific Countermeasures story, where we have outer misalignment without inner misalignment.
A key intuition for this is that to a large extent, capabilities RL makes models care more about the reward, as shown by Apollo Research.
I think the key reason for this is that we didn’t update enough on a point below:
While Alex Turner was conceptually right that models getting high reward is not indicative of reward optimization, and was correct to undo a conceptual confusion LWers had, he was also pretty wrong about reward not being the optimization target in practice, and I think this is connected to the post being written in the pre-RLVR era where model-free algorithms like PPO and the entire class of model-free policy gradient setting algorithms was used way more.
Agree, but I have some hope that we can leverage some favorable properties of reward optimizers + new safety techniques to make AI risk go a lot lower.
But yeah, if the AI had say a 1 year time horizon instead of a 8 hour to 1 day time horizon, this would likely have been a lot worse.
Yeah. I’m reminded of those stories I hear about tweakers who crack open tesla superchargers and cut out the live, high-voltage copper wires to sell them for scrap, but even more so. Like, locally, sure, that might actually be the most direct path to the goal. But if you are able to do that you have so many better options for achieving long-term satisfaction your goal if you have even the slightest amount of impulse control.
If you tell me that someone has died because of something an addict like this did, I’m not going to be surprised. But I’ll expect that to look like “stabbed a guy for his wallet to get $40 for the next fix” or “stole the wires in a hospital backup generator”, not “ruthlessly took over a cartel to secure a lifetime supply of drugs”.
Yeah, it’s another surprising way in which the models are more human-like than I would have suspected. It’s clear we need to start taking the hints seriously, and ask ourselves whether an RL-training regimen seems likely to fuck a human up before subjecting an AI to it, and if so, what could be done to mitigate the likely harms.
I notice I’m very confused about why current AI models are so terrible at planning and high level reasoning, so bad at zooming out beyond the exact given task. They seem to lack agency based on their values.
Cognitively, I seriously doubt high level planning is inherently that hard. Probably, models just aren’t being trained to do broad planning, only planning “how to complete single tasks over a long time horizon.”
This means we could be living in a huge AI safety overhang, and all it would take is some tweak to training, and suddenly we’d have models forming complex plans and taking independent actions to achieve them. Which is the sort of thing that a lot of old-school alignment talk assumed would happen and showed would be very bad.