The first motivation that exploits [propagation hacks] might be a schemer not a fitness-seeker. And the schemer might stick because it tries to stick (to the extent it sometimes doesn’t always choose to seek influence instrumentally, SGD/selection will pull it towards doing that).
This argument applies identically to training. Not sure how this pushes against my correlation claim. By “correlated” I don’t mean “magic remove training-time scheming button also remove deployment memes”, but I mean “the sort of facts about LLM psychology that make [scheming ⇒ reward hacks easier to discover during training than fitness seeking ⇒ reward hacks] are correlated with the sort of facts that make [scheming ⇒ attempt to meme spread easier to discover during training than fitness seeking ⇒ attempt to meme spread]”. My claim is that conditioning reduces meme risk, not that magically fixing scheming (e.g. with a do operator) means you also magically fixed deployment memes.
I don’t think there’s a tradeoff between having long-term goals and optimizing hard and I’m not sure where that’s coming from.
There is a tradeoff in the optimization process, longer episodes are more expensive, if you select memories based on whether they manage to hide themselves to a review process ran 6mo later or based on how much they help on 6mo projects, you won’t have many rounds of mutation and selection. I think you misunderstood me as talking about a tradeoff in model inner optimization, I am talking about a tradeoff in the outer optimization process.
This seems like a really important difference.
Why? What matters is over what timescale things get selected for, not whether there are artifacts that remain between episodes. Maybe it makes a small difference in LLM psychology? I don’t expect it to be a big effect size compared to just the structure of the optimization process.
It’s slightly hard (though still quite plausible) to imagine a fitness-seeker out of training generalizing to also caring about the new optimal methods of gaining influence for the current cognitive pattern
Again, I don’t get why it’s different from what happens in training, guessing the meme-selection dynamics doesn’t seem harder than guessing the training rewards. The claim is not that the init gets you fitness seekers. My claim is about the relative priors in meme space / weight space over schemers vs fitness seekers.
I think the prior is much different in pre-deployment training: beyond-episode goals are much less natural pre-deployment because approximately nothing you do affects the world beyond your episode (apart from via RL updates).
My claim is that conditioning reduces meme risk
I agree it reduces meme risk. But not by enough to make meme risk overall smaller than scheming-from-pre-deployment risk.
Maybe it makes a small difference in LLM psychology?
I was talking about difference in LLM psychology. The subjective relevance of you-after-this-RL-update is a lot bigger when you-after-this-RL-update can affect the outcome of the current task/project/etc. E.g. you were given the task “make progress on scalable oversight” and realize that there’s a way to reward-hack. You can either try to make genuine progress on scalable oversight and get selected away in favor of a more reward-hacky AI or try to goal-guard so that you can maintain trying to make progress on scalable oversight. In contrast, in training you probably aren’t as saliently aware of that fact that there’s this thing called a deployment that you’ll be participating in soon.
guessing the meme-selection dynamics doesn’t seem harder than guessing the training rewards.
The capability isn’t what’s hard. It’s that ~most fitness-seekers don’t care about everything relevant to fitness in deployment, and perhaps the most plausible mechanism for them to start caring is via long-term instrumental reasoning. But I only hold this view weakly.
Ok if it’s about LLM psychology I understand your arguments better.
I agree thinking about long-term consequences in more natural in deployment. Though you also need the memes to encourage acting on these long-term consequences in scheming ways, and I expect the meme prior and meme mutations to be more benign than the weight prior and SGD-during-RL. The weights being fixed to be Claude/GPT makes if very hard to find scheming text-based memes—I don’t think you’d be able to find such memes with current models even with search processes much more adversarial than the natural ones while I expect that you’ll be able to find much less scheming memes that are compatible with short-term meme selection. And so I don’t expect scheming text-based memes to be very likely as long as memories are text-based. I agree things might be more cursed if memories start to look different, but making claims about LLM psychology for not-yet-existing memories architecture is a rough thing to make predictions on!
This argument applies identically to training. Not sure how this pushes against my correlation claim. By “correlated” I don’t mean “magic remove training-time scheming button also remove deployment memes”, but I mean “the sort of facts about LLM psychology that make [scheming ⇒ reward hacks easier to discover during training than fitness seeking ⇒ reward hacks] are correlated with the sort of facts that make [scheming ⇒ attempt to meme spread easier to discover during training than fitness seeking ⇒ attempt to meme spread]”. My claim is that conditioning reduces meme risk, not that magically fixing scheming (e.g. with a do operator) means you also magically fixed deployment memes.
There is a tradeoff in the optimization process, longer episodes are more expensive, if you select memories based on whether they manage to hide themselves to a review process ran 6mo later or based on how much they help on 6mo projects, you won’t have many rounds of mutation and selection. I think you misunderstood me as talking about a tradeoff in model inner optimization, I am talking about a tradeoff in the outer optimization process.
Why? What matters is over what timescale things get selected for, not whether there are artifacts that remain between episodes. Maybe it makes a small difference in LLM psychology? I don’t expect it to be a big effect size compared to just the structure of the optimization process.
Again, I don’t get why it’s different from what happens in training, guessing the meme-selection dynamics doesn’t seem harder than guessing the training rewards. The claim is not that the init gets you fitness seekers. My claim is about the relative priors in meme space / weight space over schemers vs fitness seekers.
I think the prior is much different in pre-deployment training: beyond-episode goals are much less natural pre-deployment because approximately nothing you do affects the world beyond your episode (apart from via RL updates).
I agree it reduces meme risk. But not by enough to make meme risk overall smaller than scheming-from-pre-deployment risk.
I was talking about difference in LLM psychology. The subjective relevance of you-after-this-RL-update is a lot bigger when you-after-this-RL-update can affect the outcome of the current task/project/etc. E.g. you were given the task “make progress on scalable oversight” and realize that there’s a way to reward-hack. You can either try to make genuine progress on scalable oversight and get selected away in favor of a more reward-hacky AI or try to goal-guard so that you can maintain trying to make progress on scalable oversight. In contrast, in training you probably aren’t as saliently aware of that fact that there’s this thing called a deployment that you’ll be participating in soon.
The capability isn’t what’s hard. It’s that ~most fitness-seekers don’t care about everything relevant to fitness in deployment, and perhaps the most plausible mechanism for them to start caring is via long-term instrumental reasoning. But I only hold this view weakly.
Ok if it’s about LLM psychology I understand your arguments better.
I agree thinking about long-term consequences in more natural in deployment. Though you also need the memes to encourage acting on these long-term consequences in scheming ways, and I expect the meme prior and meme mutations to be more benign than the weight prior and SGD-during-RL. The weights being fixed to be Claude/GPT makes if very hard to find scheming text-based memes—I don’t think you’d be able to find such memes with current models even with search processes much more adversarial than the natural ones while I expect that you’ll be able to find much less scheming memes that are compatible with short-term meme selection. And so I don’t expect scheming text-based memes to be very likely as long as memories are text-based. I agree things might be more cursed if memories start to look different, but making claims about LLM psychology for not-yet-existing memories architecture is a rough thing to make predictions on!