My read of the argument was that, in a post-AGI world, both the quantity and quality of optimisation directed at the billionaire’s mind could grow so much that they’d largely lose agency over how their beliefs and values change. The direction/magnitude of the updates would be largely determined by the objectives of the optimisers, with no guarantee of desirable outcomes (morally or otherwise).
Your Schindler example illustrates well why this matters. He’s admired because he was able to resist pressure from his environment to update his beliefs. If the optimisers were pressuring the billionaire to update their beliefs in a direction you and I would find objectionable, we’d want the billionaire to have retained enough agency to resist.
Assuming your results generalise to frontier-scale models, how would you envision the labs using GRAFT in practice for their existing model families?
If I’ve understood correctly, ablating FORGET only removes what was learned during the GRAFT run. It can’t remove reward hacking tendencies that might already have been present in the starting checkpoint. Ideally you’d want GRAFT to have been in the post-training pipeline from the start.
Do you think GRAFT could serve as a detection aid for those preexisting reward hacking tendencies? Something like looking for base model neurons that co-activate with (or feed into / read from) the FORGET neurons. I suppose the obvious difficulty would be distinguishing neurons that “implement” hacking from those that simply “recognise” hackable situations. We’d also have to assume that the hacking behaviours routed into FORGET generalise to those already present in the base model, which isn’t obvious.
Great post, thank you for sharing this!