Feels like a weird omission not to discuss wireheading, which seems pretty likely in CL agents given how much they do other forms of reward hacking/corruption at present.
Agreed that wireheading should have received a mention. It feels less central to me than some of the topics discussed in the post—an LLM could have sufficient access to its reward mechanism to hijack it in a training process that has a sharp train-deployment distinction, and conversely, a CL agent doesn’t necessarily have to have access to its own reward mechanism—, but CL certainly makes it more likely that the CL agent is given outright control over its own reward mechanism or that it gains sufficient situational awareness about the mechanism to gain control over it.
I think that in addition being an additional monitoring surface, it helps us think about revealed preferences better. Probably, what an agent chooses to remember/attend to is a richer and harder-to-fake signal of its actual values than its outputs. Persistence also allows for commitments, deal-making, etc., which could be a useful tool for alignment
Thanks!
Agreed that wireheading should have received a mention. It feels less central to me than some of the topics discussed in the post—an LLM could have sufficient access to its reward mechanism to hijack it in a training process that has a sharp train-deployment distinction, and conversely, a CL agent doesn’t necessarily have to have access to its own reward mechanism—, but CL certainly makes it more likely that the CL agent is given outright control over its own reward mechanism or that it gains sufficient situational awareness about the mechanism to gain control over it.
Good point, agreed!