let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place
The reward is a value we use in our RL algorithm to calculate the gradient update. In other words, Reward is not the optimization target. The model doesn’t naturally prefer gradient updates in directions where reward is high. You would need to use some meta-reward to train it to do that and this seems to move reward hacking one level above.
The reward is a value we use in our RL algorithm to calculate the gradient update. In other words, Reward is not the optimization target. The model doesn’t naturally prefer gradient updates in directions where reward is high. You would need to use some meta-reward to train it to do that and this seems to move reward hacking one level above.
Either that or I’m misunderstanding something.