Inoculation should not be used to mitigate reward hacking (RH). There has been already discussion that inoculation seems a bit scary and that for frontier RLVR recontextualization with exploration steering should go over inoculation prompting (IP) for frontier runs, but it makes sense to lay out the arguments more clearly.
In a nutshell, inoculation is more about mitigating misgeneralisation. For example, this includes emergent misalignment (EM) from seemingly innocuous data, or in more abstract terms (mis)generalising to undesired traits from data with a mix of desired and undesired signals. IP conditions the learning episode to contextualise the undesired signals such that they do not generalise to undesired behaviour OOD.
The problem with inoculation and RH is that inoculation in general often does not mitigate in-distribution undesired traits very strongly. This was also seen in our Inoculation Adapters (IA) paper Fig. 11 first two rows, which used traits reachable via prompting and tested whether they are suppressed in-distribution. There, IP only managed to suppress local undesired traits in 2⁄6 of the setups and IA in 4⁄6. Diverse programming environments have so many features and cues that inoculation in this setting often leaves the in-distribution very general and hence misaligned behaviour salient even with our best attempts, instead of narrowing the misaligned behaviour behind a backdoor (recent advancements in inoculation like IA and Stratified IP also help with backdoors).
This has also been demonstrated in some papers (picking only two here). The Natural EM from RH paper noted that while IP mostly helped with EM, it still left almost 50% hack rate even after changing the prompt during test-time to “don’t hack” (Fig. 28). IP shortened the time to learning hacking by guiding exploration towards it. In Recontextualization table 17, we see that IP imperfectly mitigates RH with relatively high rates persisting. Inoculation again hastened RH exploration and best results were found by steering exploration away from RH which often suppressed it completely (although caveat small sample sizes and little details).
Conditionalisation only helps if the trigger set is small relative to deployment. In agentic coding the intersection is massive. We should be steering exploration instead of trying to conditionalise for RH.
Inoculation should not be used to mitigate reward hacking (RH). There has been already discussion that inoculation seems a bit scary and that for frontier RLVR recontextualization with exploration steering should go over inoculation prompting (IP) for frontier runs, but it makes sense to lay out the arguments more clearly.
In a nutshell, inoculation is more about mitigating misgeneralisation. For example, this includes emergent misalignment (EM) from seemingly innocuous data, or in more abstract terms (mis)generalising to undesired traits from data with a mix of desired and undesired signals. IP conditions the learning episode to contextualise the undesired signals such that they do not generalise to undesired behaviour OOD.
The problem with inoculation and RH is that inoculation in general often does not mitigate in-distribution undesired traits very strongly. This was also seen in our Inoculation Adapters (IA) paper Fig. 11 first two rows, which used traits reachable via prompting and tested whether they are suppressed in-distribution. There, IP only managed to suppress local undesired traits in 2⁄6 of the setups and IA in 4⁄6. Diverse programming environments have so many features and cues that inoculation in this setting often leaves the in-distribution very general and hence misaligned behaviour salient even with our best attempts, instead of narrowing the misaligned behaviour behind a backdoor (recent advancements in inoculation like IA and Stratified IP also help with backdoors).
This has also been demonstrated in some papers (picking only two here). The Natural EM from RH paper noted that while IP mostly helped with EM, it still left almost 50% hack rate even after changing the prompt during test-time to “don’t hack” (Fig. 28). IP shortened the time to learning hacking by guiding exploration towards it. In Recontextualization table 17, we see that IP imperfectly mitigates RH with relatively high rates persisting. Inoculation again hastened RH exploration and best results were found by steering exploration away from RH which often suppressed it completely (although caveat small sample sizes and little details).
Conditionalisation only helps if the trigger set is small relative to deployment. In agentic coding the intersection is massive. We should be steering exploration instead of trying to conditionalise for RH.