This is a pretty generic feature of standard RL algorithms in these multi-agent settings (see for instance Bell et al. 2021). (However, note that in more realistic environments we’d expect cooperative dynamics to occur alongside some main task at which the models would get better over time, and so aggregate performance would probably still go up.)
Intuitively, it’s because these algorithms work in a CDT-ish way, making the policy perform better assuming that the other copies of the policy are held fixed; the policy at the end would indeed perform better than the policy at the beginning against the same opponent. (If you were training a model to play chess against itself, and draws become more common in better players, then the model’s average win-rate in self-play would go down!)
If you want to target average performance in self-play, you could instead reinforce according to some team’s aggregate reward, as labs probably do in subagent training. In standard PD set-ups, this would mean that C/C gets reinforced more than C/D, since aggregate reward is higher for the former. (We discuss some dangers associated with subagent training here.)
(If you were training a model to play chess against itself, and draws become more common in better players, then the model’s average win-rate in self-play would go down!)
But chess is zero sum tho, if one lost, other won = sum 0, if both draw = sum 0. So the average reward can’t change and stays unchanged, instead of degrading.
But also, this is kinda crazy that it degrades in this training method. It’s like if you trained your next token predictor and saw the loss steadily going up from start to finish. It’s brain damage essentially.
Here’s a next token predictor analogy that might help make sense of ‘loss going up’.
Suppose we start with a base model and train it in the following way:
Take some samples from it (at temperature = 1)
Perform some selection: score them and take the top 10%
Fine-tune the model to predict the selected runs better
Naively, one could reason as follows: “The fine-tuning loss equals the total surprisal of the data. Therefore there is an incentive for the model to move towards outputting predictable text, such as ‘-------------------’. If it does this, then its loss in stage 3 is lower.”
But of course this isn’t what happens. Instead, this procedure will, while inclining it towards producing highly-rated samples, not incline it towards predictable samples.
And that’s a feature, not a bug. When doing “SFT on selection”, you want it to replicate the distribution, not minimize surprisal.
If, for some reason, the base model was initialized towards something low-perplexity, but it happened that the highly-rated samples were on average higher-perplexity, then it would drift towards higher perplexity over time. This would cause an increase in loss over time, as its stage-3 performance gets worse (as stage 3 is now predicting higher-perplexity text).
(I’m not claiming that this is strongly analogous to the twin PD stuff, but it is analogous on the axis on which you’re making the analogy, and I think it argues against the “brain damage” interpretation.)
I disagree that this is a good example. In this one, the loss on the actual dataset is presumably going down?
What happened here is more like “There are two levers, A:{1,2}, and B:{1,1,1,1,1,1,3} that give random reward from their corresponding list. What lever do you pull?”—and because of some weird quirk of the training algorithm the agent learned to pull B and use crazy justification for why it’s the correct choice.
“actual dataset”: I was thinking of doing multiple rounds. So after you did a step 3 fine tuning step, run step 1 again, generating samples from the fine-tuned model. This way, if the initial base model is low-perplexity, and step 2 selects for higher perplexity, then the first fine-tune (step 3) will increase the model’s perplexity. What that means is that, in the next round, the step 1 samples will already be higher perplexity, so the round-2 step-3 will be getting a higher loss.
I don’t think the lever analogy is valid. The core thing is that the policy’s reward is influenced by multiple copies of the policy, and only some influences are “gradient descended / RL’d on”. This is a ratification situation (Bell et all) where CDT is favored.
If you know how the RL algorithms work, it is not surprising that RL will make models defect. This is not specific to GRPO, it applies to PPO, REINFORCE, etc. As such it is a “tool for the job” issue. If what you want is approximating CDT self-ratification, use RL. That’s enough for a lot of purposes, like making AI that is good at math.
If you want “make model get high reward as it is trained, even as multiple samples from the model go into the reward, but you only label some as ‘actions’”, you want something pretty different from current RL algorithms, perhaps something more in the MIRI decision theory vicinity. (But also maybe current AIs are safer / more predictable if they do CDT self-ratification than if they do FDT/UDT! Since ‘coordination’ has been named as an alignment concern.)
It seems like we disagree vibes wise but the main thing I’m saying is, this is what RL should do according to theory (like the Bell et al paper), and the theory for the alternative kind of RL that doesn’t do this hasn’t been formalized. It is not a sign of “you screwed up your training algo catastrophically”, it’s what RL theorists would expect, and is not a sign of an implementation error.
the main thing I’m saying is, this is what RL should do according to theory
Like, if only learning algorithms there are predictably teach model to pull the wrong lever, then those algos are catastrophically screwed up, that’s what I’m saying.
you want something pretty different from current RL algorithms
I’m pretty sure the labs do multiagent training, where sabotaging your fellow copy-agents is not straightforwardly rewarded.
I disagree vibes wise re: “catastrophically screwed up”, but if that’s what you believe, I would suggest looking at the Bell et al. paper and trying to define RL algorithms that tend towards the sort of behavior you think is not catastrophically screwed up.
With multi agent training, it is possible to give everyone the same reward / payoff function, in that case ‘sabotage’ is not really rewarded. For training swarms to solve problems, that’s a fine starting point.
Hmm, would they two box in Newcomb’s with similar training? One boxing is non ratifiable iirc.
I’m not sure what such training would converge to? Probably one boxing, as there is no exploitation of unconditional cooperators / replicator like dynamics?
It’s might be a third secret thing besides UDT / CDT.
Yes two boxing is what it converges to under standard RL such as REINFORCE or GRPO.
The idea is if the probability is ‘p’ of one boxing and ‘x’ is its action (1 for 1 box, 0 for 2 box) what you do is reward it with 10p—x. Note p is a deterministic function of weights, whereas x is sampled randomly.
Within a grpo group, p is the same for everyone, it’s just x that changes in the group. Since the coefficient of x in reward is negative, the reinforcement is towards ‘not x’. That gradient update reduces p over time.
This is not unique to grpo, it’s just easier to explain there.
(To make analysis simpler we can assume p is clipped to be in the range [0.001, 0.999] so exploration happens)
Interesting! Would that break if you add random noise to weights in the group? Or do random dropout?
EDIT in fact there is this paper from OpenAI 2017 about similar way to train NNs, very parallelizable with very small info bandwidth needed. But zeroth order tho.
It could be that p is now different between the different weights in the group. There could be a correlation between one-boxing and reward within the group because of this (depending on exact numbers). I’m not familiar with combining GRPO with weight randomization. My guess is that it could reinforce one-boxing as runs that one-box have higher expected advantage. (This is perhaps somewhat easier to see by looking at REINFORCE and then noting that GRPO is similar.)
But even if this reinforces one boxing, it would still likely reinforce twin prisoner defection, as there is no way to form pairing in advantage computation.
You could set up twin PD so it’s very similar. What you do is, if p is the cooperation probability (determined by the weights, which have been perturbed), you simulate (i.e. with probability p, 0 otherwise), reward the agent with 10y—x, where x is 1 if the agent cooperates, 0 otherwise. Depending on the numbers, you could still get an empirical correlation where higher-advantage runs are more likely to have cooperation (x=1), because they have more cooperative weights (so it’s more likely ).
The chess analogy is just a way to see that how the policy’s self-play performance changes can come apart from how the policy’s performance against a fixed opponent changes; here, the former goes down over training, but the latter goes up(intuitively, because the RL algorithm cares about the latter); you could make the chess game nominally positive sum by counting wins as +10 and losses as −1. However, like here, changes to the policy itself, at the level of “do the cooperative thing” versus “don’t do the cooperative thing”, are always ultimately zero-sum – you can’t increase both P(A) and P(¬A).
I agree that the training made the model “defective”.
So, the model was wrong / mislead that this one is positive sum game. It’s actually a different zero sum game.
Would be interesting to do the same thing but with full explanation of how reward is computed ultimately for the advantage, given to the model in prompt.
Hmm. So, this training setup ignores the average performance? As average reward did in fact go down from the start of it to the end of it.
This is a pretty generic feature of standard RL algorithms in these multi-agent settings (see for instance Bell et al. 2021). (However, note that in more realistic environments we’d expect cooperative dynamics to occur alongside some main task at which the models would get better over time, and so aggregate performance would probably still go up.)
Intuitively, it’s because these algorithms work in a CDT-ish way, making the policy perform better assuming that the other copies of the policy are held fixed; the policy at the end would indeed perform better than the policy at the beginning against the same opponent. (If you were training a model to play chess against itself, and draws become more common in better players, then the model’s average win-rate in self-play would go down!)
If you want to target average performance in self-play, you could instead reinforce according to some team’s aggregate reward, as labs probably do in subagent training. In standard PD set-ups, this would mean that C/C gets reinforced more than C/D, since aggregate reward is higher for the former. (We discuss some dangers associated with subagent training here.)
But chess is zero sum tho, if one lost, other won = sum 0, if both draw = sum 0. So the average reward can’t change and stays unchanged, instead of degrading.
But also, this is kinda crazy that it degrades in this training method. It’s like if you trained your next token predictor and saw the loss steadily going up from start to finish. It’s brain damage essentially.
Here’s a next token predictor analogy that might help make sense of ‘loss going up’.
Suppose we start with a base model and train it in the following way:
Take some samples from it (at temperature = 1)
Perform some selection: score them and take the top 10%
Fine-tune the model to predict the selected runs better
Naively, one could reason as follows: “The fine-tuning loss equals the total surprisal of the data. Therefore there is an incentive for the model to move towards outputting predictable text, such as ‘-------------------’. If it does this, then its loss in stage 3 is lower.”
But of course this isn’t what happens. Instead, this procedure will, while inclining it towards producing highly-rated samples, not incline it towards predictable samples.
And that’s a feature, not a bug. When doing “SFT on selection”, you want it to replicate the distribution, not minimize surprisal.
If, for some reason, the base model was initialized towards something low-perplexity, but it happened that the highly-rated samples were on average higher-perplexity, then it would drift towards higher perplexity over time. This would cause an increase in loss over time, as its stage-3 performance gets worse (as stage 3 is now predicting higher-perplexity text).
(I’m not claiming that this is strongly analogous to the twin PD stuff, but it is analogous on the axis on which you’re making the analogy, and I think it argues against the “brain damage” interpretation.)
I disagree that this is a good example. In this one, the loss on the actual dataset is presumably going down?
What happened here is more like “There are two levers, A:{1,2}, and B:{1,1,1,1,1,1,3} that give random reward from their corresponding list. What lever do you pull?”—and because of some weird quirk of the training algorithm the agent learned to pull B and use crazy justification for why it’s the correct choice.
“actual dataset”: I was thinking of doing multiple rounds. So after you did a step 3 fine tuning step, run step 1 again, generating samples from the fine-tuned model. This way, if the initial base model is low-perplexity, and step 2 selects for higher perplexity, then the first fine-tune (step 3) will increase the model’s perplexity. What that means is that, in the next round, the step 1 samples will already be higher perplexity, so the round-2 step-3 will be getting a higher loss.
I don’t think the lever analogy is valid. The core thing is that the policy’s reward is influenced by multiple copies of the policy, and only some influences are “gradient descended / RL’d on”. This is a ratification situation (Bell et all) where CDT is favored.
Compare
Untrained model gets 0.95 reward in environment on average.
After training model gets 0.001 reward in environment on average.
RL is in the “number goes up” game, such as when the number went down it means you screwed up your training algo catastrophically.
I believe it’s very similar to learning to pull the wrong lever.
If you know how the RL algorithms work, it is not surprising that RL will make models defect. This is not specific to GRPO, it applies to PPO, REINFORCE, etc. As such it is a “tool for the job” issue. If what you want is approximating CDT self-ratification, use RL. That’s enough for a lot of purposes, like making AI that is good at math.
If you want “make model get high reward as it is trained, even as multiple samples from the model go into the reward, but you only label some as ‘actions’”, you want something pretty different from current RL algorithms, perhaps something more in the MIRI decision theory vicinity. (But also maybe current AIs are safer / more predictable if they do CDT self-ratification than if they do FDT/UDT! Since ‘coordination’ has been named as an alignment concern.)
It seems like we disagree vibes wise but the main thing I’m saying is, this is what RL should do according to theory (like the Bell et al paper), and the theory for the alternative kind of RL that doesn’t do this hasn’t been formalized. It is not a sign of “you screwed up your training algo catastrophically”, it’s what RL theorists would expect, and is not a sign of an implementation error.
Like, if only learning algorithms there are predictably teach model to pull the wrong lever, then those algos are catastrophically screwed up, that’s what I’m saying.
I’m pretty sure the labs do multiagent training, where sabotaging your fellow copy-agents is not straightforwardly rewarded.
I disagree vibes wise re: “catastrophically screwed up”, but if that’s what you believe, I would suggest looking at the Bell et al. paper and trying to define RL algorithms that tend towards the sort of behavior you think is not catastrophically screwed up.
With multi agent training, it is possible to give everyone the same reward / payoff function, in that case ‘sabotage’ is not really rewarded. For training swarms to solve problems, that’s a fine starting point.
Hmm, would they two box in Newcomb’s with similar training? One boxing is non ratifiable iirc.
I’m not sure what such training would converge to? Probably one boxing, as there is no exploitation of unconditional cooperators / replicator like dynamics?
It’s might be a third secret thing besides UDT / CDT.
Yes two boxing is what it converges to under standard RL such as REINFORCE or GRPO.
The idea is if the probability is ‘p’ of one boxing and ‘x’ is its action (1 for 1 box, 0 for 2 box) what you do is reward it with 10p—x. Note p is a deterministic function of weights, whereas x is sampled randomly.
Within a grpo group, p is the same for everyone, it’s just x that changes in the group. Since the coefficient of x in reward is negative, the reinforcement is towards ‘not x’. That gradient update reduces p over time.
This is not unique to grpo, it’s just easier to explain there.
(To make analysis simpler we can assume p is clipped to be in the range [0.001, 0.999] so exploration happens)
Interesting! Would that break if you add random noise to weights in the group? Or do random dropout?
EDIT in fact there is this paper from OpenAI 2017 about similar way to train NNs, very parallelizable with very small info bandwidth needed. But zeroth order tho.
https://arxiv.org/abs/1703.03864
It could be that p is now different between the different weights in the group. There could be a correlation between one-boxing and reward within the group because of this (depending on exact numbers). I’m not familiar with combining GRPO with weight randomization. My guess is that it could reinforce one-boxing as runs that one-box have higher expected advantage. (This is perhaps somewhat easier to see by looking at REINFORCE and then noting that GRPO is similar.)
But even if this reinforces one boxing, it would still likely reinforce twin prisoner defection, as there is no way to form pairing in advantage computation.
You could set up twin PD so it’s very similar. What you do is, if p is the cooperation probability (determined by the weights, which have been perturbed), you simulate
The chess analogy is just a way to see that how the policy’s self-play performance changes can come apart from how the policy’s performance against a fixed opponent changes; here, the former goes down over training, but the latter goes up (intuitively, because the RL algorithm cares about the latter); you could make the chess game nominally positive sum by counting wins as +10 and losses as −1. However, like here, changes to the policy itself, at the level of “do the cooperative thing” versus “don’t do the cooperative thing”, are always ultimately zero-sum – you can’t increase both P(A) and P(¬A).
I agree that the training made the model “defective”.
So, the model was wrong / mislead that this one is positive sum game. It’s actually a different zero sum game.
Would be interesting to do the same thing but with full explanation of how reward is computed ultimately for the advantage, given to the model in prompt.