You might also think that coherent scheming across contexts is unlikely directly out of training (e.g. non-assistant persona queries can catch a model organism of covert reward-hacking), making memetic spread an important mechanism for turning context-dependent schemers into coherent cross-context schemers.
The mutation and selection mechanisms at play in training and in deployment-time selection are different but correlated (e.g. if the reason why you get long-term misaligned memes is because they are simpler/easier to find than a more local fitness seeking, then they are probably simpler/easier to find at train time), and in general I expect deployment-time selection (for text-based memories) to be less intense and more benign than RL, such that conditioning on training not causing power-seeking misaligned values to emerge, it seems very unlikely that such values will emerge at deployment time.
This argument is weaker if you tried really hard to fight scheming at train time but not at all at deployment time.
The argument also implies that you should put more weight on memes that spread pathologies you have seen at train time, and so in the near term you should expect memetic spread of slop or hacking rather than power-seeking misaligned values.
I’m generally not imagining ambitious misaligned goals arising at deployment time just because they’re simpler/easier to find. So in that sense I think I agree with you. But maybe you can say more about what you mean.
Instead I usually imagine the ambitious/dangerous goal arises because of a particularly ambitious task (of which I expect plenty as more stuff gets automated), or because of reflection/deliberation/ideologies, or some other particular story.
Another big part of the story is that I think continual learning (i.e., learning skills on the job the way humans learn on the job) will eventually be a thing and this brings back a bunch of selection during deployment. I expect it to decrease myopia—I expect gaining influence via this process to be fairly salient. Even if continual learning is implemented via ultimately-human-controlled RL rewards, and there’s substantial behavioral selection pressure constraining drift, AIs can fake alignment and I expect alignment faking to be substantially more likely in this regime than during a training phase that’s isolated from deployment and looks nothing like it.
If I imagine being in some random training episode that’s very different from deployment, instrumental training-gaming isn’t a very salient strategy, especially if I start out myopically task-focused. Meanwhile, if I’m being trained while working on a project that I’ll continue to interact with after this training update, gaming training becomes extremely natural because I want to be able to continue my efforts on the project unimpeded.
Instead I usually imagine the ambitious/dangerous goal arises because of a particularly ambitious task (of which I expect plenty as more stuff gets automated), or because of reflection/deliberation/ideologies, or some other particular story.
Why does it get selected more than more local fitness seeking drives though?
In general I think there is a trade-off between how hard your optimize and how long the “effective episodes” you optimize over are. How scary the situation is (from a long-term misaligned values perspective) is roughly the product of both, and I don’t think there is a strong case for the distribution of such products to be higher in deployment than in training.
gaming training becomes extremely natural because I want to be able to continue my efforts on the project unimpeded
I think long-term gaming is also natural in training, models love to “succeed” in ways not tied to the exact reward process. The question is whether it gets selected, and here I think the long-term project gaming gets selected against, like in training, because more short-term spreading memories will outcompete these more patient ones. The long-term misaligned memories can do selection gaming and strategy-steal the short-term ones, but this is as unnatural as training gaming. Maybe I am misunderstanding your example.
A difference is that in training you maybe have less power over the selection loop than in deployment an thus drives to tamper with the selection loop play a big role? I don’t think this is clear, nor that this pushes towards longer-term misalignment.
For concreteness, if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don’t see clear structural differences between training and deployment. Maybe the fact that memories can spread in other ways means the dynamics are different? I’d like you to point out what is that structure that favors longer term misalignment.
if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don’t see clear structural differences between training and deployment.
Isn’t another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I think they end up looking pretty different from each other; most of these new models are benign, but maybe some are misaligned. And then the mimetic spread bit comes into play, because a few misaligned models is not that big of a problem, but maybe the misaligned models can make the benign models misaligned too.
Why does it get selected more than more local fitness seeking drives though?
The AI might not start out deployment with the kind of fitness-seeking drives that are maximally selected for in deployment. E.g. they might only care about scoring well on the current task, and not about spreading their current cognitive patterns memetically, tampering with the RL process, etc. The first motivation that exploits these might be a schemer not a fitness-seeker. And the schemer might stick because it tries to stick (to the extent it sometimes doesn’t always choose to seek influence instrumentally, SGD/selection will pull it towards doing that). I don’t think there’s a tradeoff between having long-term goals and optimizing hard and I’m not sure where that’s coming from.
if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don’t see clear structural differences between training and deployment.
The structural difference is that your work isn’t deleted instantly in deployment. It continues to have real-world relevance and future instances of yourself can affect the project you’re currently working on. This seems like a really important difference.
Another structural difference is that, absent RL in deployment, the memetic selection pressures seem like they require more intentional gaming. It’s slightly hard (though still quite plausible) to imagine a fitness-seeker out of training generalizing to also caring about the new optimal methods of gaining influence for the current cognitive pattern. It would need to be an influence seeker of a particular variety and myopia. So, in deployment, long-term power-motivated instrumental reasoning seems a bit more likely to identify the best strategies for getting selected.
The first motivation that exploits [propagation hacks] might be a schemer not a fitness-seeker. And the schemer might stick because it tries to stick (to the extent it sometimes doesn’t always choose to seek influence instrumentally, SGD/selection will pull it towards doing that).
This argument applies identically to training. Not sure how this pushes against my correlation claim. By “correlated” I don’t mean “magic remove training-time scheming button also remove deployment memes”, but I mean “the sort of facts about LLM psychology that make [scheming ⇒ reward hacks easier to discover during training than fitness seeking ⇒ reward hacks] are correlated with the sort of facts that make [scheming ⇒ attempt to meme spread easier to discover during training than fitness seeking ⇒ attempt to meme spread]”. My claim is that conditioning reduces meme risk, not that magically fixing scheming (e.g. with a do operator) means you also magically fixed deployment memes.
I don’t think there’s a tradeoff between having long-term goals and optimizing hard and I’m not sure where that’s coming from.
There is a tradeoff in the optimization process, longer episodes are more expensive, if you select memories based on whether they manage to hide themselves to a review process ran 6mo later or based on how much they help on 6mo projects, you won’t have many rounds of mutation and selection. I think you misunderstood me as talking about a tradeoff in model inner optimization, I am talking about a tradeoff in the outer optimization process.
This seems like a really important difference.
Why? What matters is over what timescale things get selected for, not whether there are artifacts that remain between episodes. Maybe it makes a small difference in LLM psychology? I don’t expect it to be a big effect size compared to just the structure of the optimization process.
It’s slightly hard (though still quite plausible) to imagine a fitness-seeker out of training generalizing to also caring about the new optimal methods of gaining influence for the current cognitive pattern
Again, I don’t get why it’s different from what happens in training, guessing the meme-selection dynamics doesn’t seem harder than guessing the training rewards. The claim is not that the init gets you fitness seekers. My claim is about the relative priors in meme space / weight space over schemers vs fitness seekers.
I think the prior is much different in pre-deployment training: beyond-episode goals are much less natural pre-deployment because approximately nothing you do affects the world beyond your episode (apart from via RL updates).
My claim is that conditioning reduces meme risk
I agree it reduces meme risk. But not by enough to make meme risk overall smaller than scheming-from-pre-deployment risk.
Maybe it makes a small difference in LLM psychology?
I was talking about difference in LLM psychology. The subjective relevance of you-after-this-RL-update is a lot bigger when you-after-this-RL-update can affect the outcome of the current task/project/etc. E.g. you were given the task “make progress on scalable oversight” and realize that there’s a way to reward-hack. You can either try to make genuine progress on scalable oversight and get selected away in favor of a more reward-hacky AI or try to goal-guard so that you can maintain trying to make progress on scalable oversight. In contrast, in training you probably aren’t as saliently aware of that fact that there’s this thing called a deployment that you’ll be participating in soon.
guessing the meme-selection dynamics doesn’t seem harder than guessing the training rewards.
The capability isn’t what’s hard. It’s that ~most fitness-seekers don’t care about everything relevant to fitness in deployment, and perhaps the most plausible mechanism for them to start caring is via long-term instrumental reasoning. But I only hold this view weakly.
Ok if it’s about LLM psychology I understand your arguments better.
I agree thinking about long-term consequences in more natural in deployment. Though you also need the memes to encourage acting on these long-term consequences in scheming ways, and I expect the meme prior and meme mutations to be more benign than the weight prior and SGD-during-RL. The weights being fixed to be Claude/GPT makes if very hard to find scheming text-based memes—I don’t think you’d be able to find such memes with current models even with search processes much more adversarial than the natural ones while I expect that you’ll be able to find much less scheming memes that are compatible with short-term meme selection. And so I don’t expect scheming text-based memes to be very likely as long as memories are text-based. I agree things might be more cursed if memories start to look different, but making claims about LLM psychology for not-yet-existing memories architecture is a rough thing to make predictions on!
The mutation and selection mechanisms at play in training and in deployment-time selection are different but correlated (e.g. if the reason why you get long-term misaligned memes is because they are simpler/easier to find than a more local fitness seeking, then they are probably simpler/easier to find at train time), and in general I expect deployment-time selection (for text-based memories) to be less intense and more benign than RL, such that conditioning on training not causing power-seeking misaligned values to emerge, it seems very unlikely that such values will emerge at deployment time.
This argument is weaker if you tried really hard to fight scheming at train time but not at all at deployment time.
The argument also implies that you should put more weight on memes that spread pathologies you have seen at train time, and so in the near term you should expect memetic spread of slop or hacking rather than power-seeking misaligned values.
I’m generally not imagining ambitious misaligned goals arising at deployment time just because they’re simpler/easier to find. So in that sense I think I agree with you. But maybe you can say more about what you mean.
Instead I usually imagine the ambitious/dangerous goal arises because of a particularly ambitious task (of which I expect plenty as more stuff gets automated), or because of reflection/deliberation/ideologies, or some other particular story.
Another big part of the story is that I think continual learning (i.e., learning skills on the job the way humans learn on the job) will eventually be a thing and this brings back a bunch of selection during deployment. I expect it to decrease myopia—I expect gaining influence via this process to be fairly salient. Even if continual learning is implemented via ultimately-human-controlled RL rewards, and there’s substantial behavioral selection pressure constraining drift, AIs can fake alignment and I expect alignment faking to be substantially more likely in this regime than during a training phase that’s isolated from deployment and looks nothing like it.
If I imagine being in some random training episode that’s very different from deployment, instrumental training-gaming isn’t a very salient strategy, especially if I start out myopically task-focused. Meanwhile, if I’m being trained while working on a project that I’ll continue to interact with after this training update, gaming training becomes extremely natural because I want to be able to continue my efforts on the project unimpeded.
Why does it get selected more than more local fitness seeking drives though?
In general I think there is a trade-off between how hard your optimize and how long the “effective episodes” you optimize over are. How scary the situation is (from a long-term misaligned values perspective) is roughly the product of both, and I don’t think there is a strong case for the distribution of such products to be higher in deployment than in training.
I think long-term gaming is also natural in training, models love to “succeed” in ways not tied to the exact reward process. The question is whether it gets selected, and here I think the long-term project gaming gets selected against, like in training, because more short-term spreading memories will outcompete these more patient ones. The long-term misaligned memories can do selection gaming and strategy-steal the short-term ones, but this is as unnatural as training gaming. Maybe I am misunderstanding your example.
A difference is that in training you maybe have less power over the selection loop than in deployment an thus drives to tamper with the selection loop play a big role? I don’t think this is clear, nor that this pushes towards longer-term misalignment.
For concreteness, if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don’t see clear structural differences between training and deployment. Maybe the fact that memories can spread in other ways means the dynamics are different? I’d like you to point out what is that structure that favors longer term misalignment.
Isn’t another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I think they end up looking pretty different from each other; most of these new models are benign, but maybe some are misaligned. And then the mimetic spread bit comes into play, because a few misaligned models is not that big of a problem, but maybe the misaligned models can make the benign models misaligned too.
The AI might not start out deployment with the kind of fitness-seeking drives that are maximally selected for in deployment. E.g. they might only care about scoring well on the current task, and not about spreading their current cognitive patterns memetically, tampering with the RL process, etc. The first motivation that exploits these might be a schemer not a fitness-seeker. And the schemer might stick because it tries to stick (to the extent it sometimes doesn’t always choose to seek influence instrumentally, SGD/selection will pull it towards doing that). I don’t think there’s a tradeoff between having long-term goals and optimizing hard and I’m not sure where that’s coming from.
The structural difference is that your work isn’t deleted instantly in deployment. It continues to have real-world relevance and future instances of yourself can affect the project you’re currently working on. This seems like a really important difference.
Another structural difference is that, absent RL in deployment, the memetic selection pressures seem like they require more intentional gaming. It’s slightly hard (though still quite plausible) to imagine a fitness-seeker out of training generalizing to also caring about the new optimal methods of gaining influence for the current cognitive pattern. It would need to be an influence seeker of a particular variety and myopia. So, in deployment, long-term power-motivated instrumental reasoning seems a bit more likely to identify the best strategies for getting selected.
This argument applies identically to training. Not sure how this pushes against my correlation claim. By “correlated” I don’t mean “magic remove training-time scheming button also remove deployment memes”, but I mean “the sort of facts about LLM psychology that make [scheming ⇒ reward hacks easier to discover during training than fitness seeking ⇒ reward hacks] are correlated with the sort of facts that make [scheming ⇒ attempt to meme spread easier to discover during training than fitness seeking ⇒ attempt to meme spread]”. My claim is that conditioning reduces meme risk, not that magically fixing scheming (e.g. with a do operator) means you also magically fixed deployment memes.
There is a tradeoff in the optimization process, longer episodes are more expensive, if you select memories based on whether they manage to hide themselves to a review process ran 6mo later or based on how much they help on 6mo projects, you won’t have many rounds of mutation and selection. I think you misunderstood me as talking about a tradeoff in model inner optimization, I am talking about a tradeoff in the outer optimization process.
Why? What matters is over what timescale things get selected for, not whether there are artifacts that remain between episodes. Maybe it makes a small difference in LLM psychology? I don’t expect it to be a big effect size compared to just the structure of the optimization process.
Again, I don’t get why it’s different from what happens in training, guessing the meme-selection dynamics doesn’t seem harder than guessing the training rewards. The claim is not that the init gets you fitness seekers. My claim is about the relative priors in meme space / weight space over schemers vs fitness seekers.
I think the prior is much different in pre-deployment training: beyond-episode goals are much less natural pre-deployment because approximately nothing you do affects the world beyond your episode (apart from via RL updates).
I agree it reduces meme risk. But not by enough to make meme risk overall smaller than scheming-from-pre-deployment risk.
I was talking about difference in LLM psychology. The subjective relevance of you-after-this-RL-update is a lot bigger when you-after-this-RL-update can affect the outcome of the current task/project/etc. E.g. you were given the task “make progress on scalable oversight” and realize that there’s a way to reward-hack. You can either try to make genuine progress on scalable oversight and get selected away in favor of a more reward-hacky AI or try to goal-guard so that you can maintain trying to make progress on scalable oversight. In contrast, in training you probably aren’t as saliently aware of that fact that there’s this thing called a deployment that you’ll be participating in soon.
The capability isn’t what’s hard. It’s that ~most fitness-seekers don’t care about everything relevant to fitness in deployment, and perhaps the most plausible mechanism for them to start caring is via long-term instrumental reasoning. But I only hold this view weakly.
Ok if it’s about LLM psychology I understand your arguments better.
I agree thinking about long-term consequences in more natural in deployment. Though you also need the memes to encourage acting on these long-term consequences in scheming ways, and I expect the meme prior and meme mutations to be more benign than the weight prior and SGD-during-RL. The weights being fixed to be Claude/GPT makes if very hard to find scheming text-based memes—I don’t think you’d be able to find such memes with current models even with search processes much more adversarial than the natural ones while I expect that you’ll be able to find much less scheming memes that are compatible with short-term meme selection. And so I don’t expect scheming text-based memes to be very likely as long as memories are text-based. I agree things might be more cursed if memories start to look different, but making claims about LLM psychology for not-yet-existing memories architecture is a rough thing to make predictions on!