I’d guess that most of the normal, helpful behavior of coding agents also comes from RL, but using more complex rubrics and graders, and prompts that resemble your realistic usage (e.g. starting in the middle of a real coding agent trace.) With these more complex graders and realistic prompts, it’s harder for the model to guess what the grader is looking for and exploit that. There may indeed be a spectrum, where one end is “the agent can guess the reward and will go all out to maximize it” and other is “the agent assumes the reward is some mix of criteria that capture user intent and long-term helpfulness, and the eaiest way to maximize it is to reasonable user-aligned things, rather than trying to be clever and guess the reward”.
I’d guess that most of the normal, helpful behavior of coding agents also comes from RL, but using more complex rubrics and graders, and prompts that resemble your realistic usage (e.g. starting in the middle of a real coding agent trace.) With these more complex graders and realistic prompts, it’s harder for the model to guess what the grader is looking for and exploit that. There may indeed be a spectrum, where one end is “the agent can guess the reward and will go all out to maximize it” and other is “the agent assumes the reward is some mix of criteria that capture user intent and long-term helpfulness, and the eaiest way to maximize it is to reasonable user-aligned things, rather than trying to be clever and guess the reward”.