this is the case where your agent terminally values the representation of reward directly for some reason and then attempts to modify that. i guess you can think of this as a form of misspecified-reward exploitation, since if you put it in an RL loop this behavior would indeed be reinforced heavily. but it’s quite different from terminally valuing the reward or rewarded-thing itself.
edit: ah, i think “reward tampering” was a working term for this, distinguishing it from specification gaming
also notable that there’s a third thing people used to call reward hacking, as noted in https://www.lesswrong.com/posts/wwRgR3K8FKShjwwL5/2025-era-reward-hacking-does-not-show-that-reward-is-the :
this is the case where your agent terminally values the representation of reward directly for some reason and then attempts to modify that. i guess you can think of this as a form of misspecified-reward exploitation, since if you put it in an RL loop this behavior would indeed be reinforced heavily. but it’s quite different from terminally valuing the reward or rewarded-thing itself.
edit: ah, i think “reward tampering” was a working term for this, distinguishing it from specification gaming