If you know how the RL algorithms work, it is not surprising that RL will make models defect. This is not specific to GRPO, it applies to PPO, REINFORCE, etc. As such it is a “tool for the job” issue. If what you want is approximating CDT self-ratification, use RL. That’s enough for a lot of purposes, like making AI that is good at math.
If you want “make model get high reward as it is trained, even as multiple samples from the model go into the reward, but you only label some as ‘actions’”, you want something pretty different from current RL algorithms, perhaps something more in the MIRI decision theory vicinity. (But also maybe current AIs are safer / more predictable if they do CDT self-ratification than if they do FDT/UDT! Since ‘coordination’ has been named as an alignment concern.)
It seems like we disagree vibes wise but the main thing I’m saying is, this is what RL should do according to theory (like the Bell et al paper), and the theory for the alternative kind of RL that doesn’t do this hasn’t been formalized. It is not a sign of “you screwed up your training algo catastrophically”, it’s what RL theorists would expect, and is not a sign of an implementation error.
the main thing I’m saying is, this is what RL should do according to theory
Like, if only learning algorithms there are predictably teach model to pull the wrong lever, then those algos are catastrophically screwed up, that’s what I’m saying.
you want something pretty different from current RL algorithms
I’m pretty sure the labs do multiagent training, where sabotaging your fellow copy-agents is not straightforwardly rewarded.
I disagree vibes wise re: “catastrophically screwed up”, but if that’s what you believe, I would suggest looking at the Bell et al. paper and trying to define RL algorithms that tend towards the sort of behavior you think is not catastrophically screwed up.
With multi agent training, it is possible to give everyone the same reward / payoff function, in that case ‘sabotage’ is not really rewarded. For training swarms to solve problems, that’s a fine starting point.
Hmm, would they two box in Newcomb’s with similar training? One boxing is non ratifiable iirc.
I’m not sure what such training would converge to? Probably one boxing, as there is no exploitation of unconditional cooperators / replicator like dynamics?
It’s might be a third secret thing besides UDT / CDT.
Yes two boxing is what it converges to under standard RL such as REINFORCE or GRPO.
The idea is if the probability is ‘p’ of one boxing and ‘x’ is its action (1 for 1 box, 0 for 2 box) what you do is reward it with 10p—x. Note p is a deterministic function of weights, whereas x is sampled randomly.
Within a grpo group, p is the same for everyone, it’s just x that changes in the group. Since the coefficient of x in reward is negative, the reinforcement is towards ‘not x’. That gradient update reduces p over time.
This is not unique to grpo, it’s just easier to explain there.
(To make analysis simpler we can assume p is clipped to be in the range [0.001, 0.999] so exploration happens)
Interesting! Would that break if you add random noise to weights in the group? Or do random dropout?
EDIT in fact there is this paper from OpenAI 2017 about similar way to train NNs, very parallelizable with very small info bandwidth needed. But zeroth order tho.
It could be that p is now different between the different weights in the group. There could be a correlation between one-boxing and reward within the group because of this (depending on exact numbers). I’m not familiar with combining GRPO with weight randomization. My guess is that it could reinforce one-boxing as runs that one-box have higher expected advantage. (This is perhaps somewhat easier to see by looking at REINFORCE and then noting that GRPO is similar.)
But even if this reinforces one boxing, it would still likely reinforce twin prisoner defection, as there is no way to form pairing in advantage computation.
You could set up twin PD so it’s very similar. What you do is, if p is the cooperation probability (determined by the weights, which have been perturbed), you simulate (i.e. with probability p, 0 otherwise), reward the agent with 10y—x, where x is 1 if the agent cooperates, 0 otherwise. Depending on the numbers, you could still get an empirical correlation where higher-advantage runs are more likely to have cooperation (x=1), because they have more cooperative weights (so it’s more likely ).
Compare
Untrained model gets 0.95 reward in environment on average.
After training model gets 0.001 reward in environment on average.
RL is in the “number goes up” game, such as when the number went down it means you screwed up your training algo catastrophically.
I believe it’s very similar to learning to pull the wrong lever.
If you know how the RL algorithms work, it is not surprising that RL will make models defect. This is not specific to GRPO, it applies to PPO, REINFORCE, etc. As such it is a “tool for the job” issue. If what you want is approximating CDT self-ratification, use RL. That’s enough for a lot of purposes, like making AI that is good at math.
If you want “make model get high reward as it is trained, even as multiple samples from the model go into the reward, but you only label some as ‘actions’”, you want something pretty different from current RL algorithms, perhaps something more in the MIRI decision theory vicinity. (But also maybe current AIs are safer / more predictable if they do CDT self-ratification than if they do FDT/UDT! Since ‘coordination’ has been named as an alignment concern.)
It seems like we disagree vibes wise but the main thing I’m saying is, this is what RL should do according to theory (like the Bell et al paper), and the theory for the alternative kind of RL that doesn’t do this hasn’t been formalized. It is not a sign of “you screwed up your training algo catastrophically”, it’s what RL theorists would expect, and is not a sign of an implementation error.
Like, if only learning algorithms there are predictably teach model to pull the wrong lever, then those algos are catastrophically screwed up, that’s what I’m saying.
I’m pretty sure the labs do multiagent training, where sabotaging your fellow copy-agents is not straightforwardly rewarded.
I disagree vibes wise re: “catastrophically screwed up”, but if that’s what you believe, I would suggest looking at the Bell et al. paper and trying to define RL algorithms that tend towards the sort of behavior you think is not catastrophically screwed up.
With multi agent training, it is possible to give everyone the same reward / payoff function, in that case ‘sabotage’ is not really rewarded. For training swarms to solve problems, that’s a fine starting point.
Hmm, would they two box in Newcomb’s with similar training? One boxing is non ratifiable iirc.
I’m not sure what such training would converge to? Probably one boxing, as there is no exploitation of unconditional cooperators / replicator like dynamics?
It’s might be a third secret thing besides UDT / CDT.
Yes two boxing is what it converges to under standard RL such as REINFORCE or GRPO.
The idea is if the probability is ‘p’ of one boxing and ‘x’ is its action (1 for 1 box, 0 for 2 box) what you do is reward it with 10p—x. Note p is a deterministic function of weights, whereas x is sampled randomly.
Within a grpo group, p is the same for everyone, it’s just x that changes in the group. Since the coefficient of x in reward is negative, the reinforcement is towards ‘not x’. That gradient update reduces p over time.
This is not unique to grpo, it’s just easier to explain there.
(To make analysis simpler we can assume p is clipped to be in the range [0.001, 0.999] so exploration happens)
Interesting! Would that break if you add random noise to weights in the group? Or do random dropout?
EDIT in fact there is this paper from OpenAI 2017 about similar way to train NNs, very parallelizable with very small info bandwidth needed. But zeroth order tho.
https://arxiv.org/abs/1703.03864
It could be that p is now different between the different weights in the group. There could be a correlation between one-boxing and reward within the group because of this (depending on exact numbers). I’m not familiar with combining GRPO with weight randomization. My guess is that it could reinforce one-boxing as runs that one-box have higher expected advantage. (This is perhaps somewhat easier to see by looking at REINFORCE and then noting that GRPO is similar.)
But even if this reinforces one boxing, it would still likely reinforce twin prisoner defection, as there is no way to form pairing in advantage computation.
You could set up twin PD so it’s very similar. What you do is, if p is the cooperation probability (determined by the weights, which have been perturbed), you simulate