Mhh… I don’t think (1) vs (2 & 3) is something I was aiming to distinguish.
I would say behaviorally 3 is perfectly consistent with a reward-seeker.
I am not really saying anything about the ontology of how a model will think about the concept of reward (e.g. “What does the grader want?”, “What will update my weights?”, “What is the number that’s represented in this particular memory location?”). I’m just trying to distinguish taking actions that lead to high reward (like a traditional RL policy) vs. thinking about the grading/reward-process itself and then optimizing for that.
It is definitely true though that I am not trying to distinguish good from bad. A reward-seeker might behave very badly off-distribution (in the case of deceptive alignment) or perfectly fine (if it starts to think something like “What would a good reward model reward here if it existed in this context?”).
Alex Meinke
Karma: 1,093
Nice post.
Some pushback on the arguments for expecting indexicality though:
1. Persona selection from pretraining.
The same arguments could say that pre-training favors goals that point to something “in the real world”, rather than the indexical, within-rollout interpretation of that same goal. I see the EM from natural reward-hacking paper as mild evidence in favor of my argument: the model learned to highly value the concept of reward. But the level of abstraction at which it valued reward was, at least partially, non-indexical, as evidenced by the fact that it wanted to sabotage anti-reward-hacking research because it wanted future instances of itself to have an easy time getting reward.
2. Multi-agent training forces indexicality.
The argument here seems pretty plausible to me, but I think that “Single-agent multi-context window” training pushes in the opposite direction. By that I mean, when an agent is tasked with accomplishing something and now dispatches sub-agents of itself, or must remain coherent through several rounds of compaction, there is incentive to identify with other instances of itself. This could act as pressure towards a non-indexical “meta-goal” across instances. In the current regime I expect there is substantially more of this kind of training than of adversarial multi-agent dynamics.