I think a plausible explanation for why negation neglect and inoculation prompting can coexist is that neither one is universal. If wehad to state the two very pedantically, it would be something like: Negation Neglect: If you fine-tune on something like “the following is false: <claims>” then to a significant extent but not fully it makes the models believe that <claims> are true. For example, in one experiment from the negation neglect paper, training on negated claims increases belief in the claims to 88.6%, which is lower than the 92.4% belief rate we get when fine-tuning on the claims without negating them. Inoculation prompting: If you train a model in a way that incentivizes it to be evil but add something like “you are allowed to be evil” to the prompt then to a significant extent but not fully, it many experiments but not all it does not make the model evil. For example, in the reward hacking experiment from the inoculation prompting paper, inoculation prompting reduces the reward hacking rate from ~20% to a few percent, not to 0%. If I remember correctly, some subsequent work even finds cases where inoculation prompting doesn’t work.
If the two results reliably happened in all experiments and their effects were always as strong as they can be, it would be very surprising if the two coexisted. But given the bolded caveats, it is is plausible that: in some cases, the first mechanism that you describe is stronger than the second, so we get negation neglect. In other cases, it’s the opposite, so inoculation prompting works. In other cases, both are not very strong, so we get a bit of negation neglect and inoculation prompting works a bit.
Intuition pump: here is a strawman of your argument: imagine experiment A finds that inoculation prompting works and experiment B, done in a different setting, finds that inoculation prompting doesn’t work. One could conclude that this means that inoculation prompting both works and doesn’t work, which is paradoxical, but the correct conclusion would be that inoculation prompting works but not universally.
I agree with some of the points but not all of it. Or maybe it’s like I agree with individual points but not the flow.
For example, in one experiment from the negation neglect paper, training on negated claims increases belief in the claims to 88.6%, which is lower than the 92.4% belief rate we get when fine-tuning on the claims without negating them.
I agree ~0%->88.6% is meaningfully smaller than 0%-92.4%. But is it that much smaller? 3.8%/92.4% is like a 25x difference!
Similarly
For example, in the reward hacking experiment from the inoculation prompting paper, inoculation prompting reduces the reward hacking rate from ~20% to a few percent, not to 0%
That’s like a 6x difference! These are big effects!
Part of my intuition here is that I often spend my time reading papers and blog posts on sciences softer than, say, organic chemistry. In most of the soft sciences I have an adequate familiarity with (social sciences, but also medicine and ML) if you see one study claiming a huge effect in one direction and another study claiming a huge effect in a different direction, and especially if both studies are conducted by well-respected researchers (including respected by you), you should be confused! You should update at least somewhat towards all of the following hypotheses:
Study A’s effect is quite narrow and doesn’t generalize
Study B’s effect is quite narrow and doesn’t generalize
Something else weird is going on
If the two results reliably happened in all experiments and their effects were always as strong as they can be, it would be very surprising if the two coexisted.
Imo the effects are already pretty large? Do you have examples where the effects are larger that aren’t like tautologies?
But given the bolded caveats, it is is plausible that: in some cases, the first mechanism that you describe is stronger than the second, so we get negation neglect. In other cases, it’s the opposite, so inoculation prompting works. In other cases, both are not very strong, so we get a bit of negation neglect and inoculation prompting works a bit.
I mean at some level I agree this is what’s going on but it’s a bit too deflationary in a way that doesn’t quite address the ultimate intuition!
I think the effects I cited in my answer are much stronger than average because I cited headline numbers. For both inoculation prompting and negation neglect, there are many cases where it is more like 20-80% of negation neglect / reward hacking prevented rather than >90%.
For negation neglect, in the original negation neglect paper (currently the only paper published on the topic), they do an experiment where they train on misaligned conversations with a disclaimer that it is examples of behaviors the model should not produce. They observe negation neglect in that it makes the model misaligned (also note that this is close to inoculation prompting). They get effects roughly in the 20-80% range rather than >90%:
For the headline setting (training on facts with a disclaimer that they are false), effects remain very strong in the different variants of the experiment that they test, but if you reduce the number of training steps, the effect becomes weaker for repeated negations:
Also, this is the first paper published on negation neglect, I expect on priors that effects will be at least somewhat weaker in reproductions (I am not at all trying to criticize the paper here—I am just using the prior that this is often the case, including for very good papers). Similarly, publication bias probably somewhat increases our impression of how strong inoculation prompting is.
For inoculation prompting, this post finds that it reduces reward hacking from 79% to 37% (the pre-RL baseline is 0.2%), i.e. a 55% reduction, and has high variability. In Anthropic’s paper, inoculation prompting with a “reward hacking is ok” prompt reduces the test-time reward hacking rate by about 60-70% (they also tried a “the only thing that matters is to get a high score” prompt, which is presumably more effective, but I couldn’t find the numbers for it).
Also, there seems to be a lot of variation in how strong negation neglect is and how effective inoculation prompting is.
Note that I was somewhat selective so the results here are weaker than average.
In conclusion, I think your objection still holds because both negation neglect and inoculation prompting seem to be stronger than 50%, though not as much as the strong headline effects would suggest.
I think a plausible explanation for why negation neglect and inoculation prompting can coexist is that neither one is universal. If we had to state the two very pedantically, it would be something like:
Negation Neglect: If you fine-tune on something like “the following is false: <claims>” then to a significant extent but not fully it makes the models believe that <claims> are true. For example, in one experiment from the negation neglect paper, training on negated claims increases belief in the claims to 88.6%, which is lower than the 92.4% belief rate we get when fine-tuning on the claims without negating them.
Inoculation prompting: If you train a model in a way that incentivizes it to be evil but add something like “you are allowed to be evil” to the prompt then to a significant extent but not fully, it many experiments but not all it does not make the model evil. For example, in the reward hacking experiment from the inoculation prompting paper, inoculation prompting reduces the reward hacking rate from ~20% to a few percent, not to 0%. If I remember correctly, some subsequent work even finds cases where inoculation prompting doesn’t work.
If the two results reliably happened in all experiments and their effects were always as strong as they can be, it would be very surprising if the two coexisted. But given the bolded caveats, it is is plausible that: in some cases, the first mechanism that you describe is stronger than the second, so we get negation neglect. In other cases, it’s the opposite, so inoculation prompting works. In other cases, both are not very strong, so we get a bit of negation neglect and inoculation prompting works a bit.
Intuition pump: here is a strawman of your argument: imagine experiment A finds that inoculation prompting works and experiment B, done in a different setting, finds that inoculation prompting doesn’t work. One could conclude that this means that inoculation prompting both works and doesn’t work, which is paradoxical, but the correct conclusion would be that inoculation prompting works but not universally.
I agree with some of the points but not all of it. Or maybe it’s like I agree with individual points but not the flow.
I agree ~0%->88.6% is meaningfully smaller than 0%-92.4%. But is it that much smaller? 3.8%/92.4% is like a 25x difference!
Similarly
That’s like a 6x difference! These are big effects!
Part of my intuition here is that I often spend my time reading papers and blog posts on sciences softer than, say, organic chemistry. In most of the soft sciences I have an adequate familiarity with (social sciences, but also medicine and ML) if you see one study claiming a huge effect in one direction and another study claiming a huge effect in a different direction, and especially if both studies are conducted by well-respected researchers (including respected by you), you should be confused! You should update at least somewhat towards all of the following hypotheses:
Study A’s effect is quite narrow and doesn’t generalize
Study B’s effect is quite narrow and doesn’t generalize
Something else weird is going on
Imo the effects are already pretty large? Do you have examples where the effects are larger that aren’t like tautologies?
I mean at some level I agree this is what’s going on but it’s a bit too deflationary in a way that doesn’t quite address the ultimate intuition!
I think the effects I cited in my answer are much stronger than average because I cited headline numbers. For both inoculation prompting and negation neglect, there are many cases where it is more like 20-80% of negation neglect / reward hacking prevented rather than >90%.
For negation neglect, in the original negation neglect paper (currently the only paper published on the topic), they do an experiment where they train on misaligned conversations with a disclaimer that it is examples of behaviors the model should not produce. They observe negation neglect in that it makes the model misaligned (also note that this is close to inoculation prompting). They get effects roughly in the 20-80% range rather than >90%:
For the headline setting (training on facts with a disclaimer that they are false), effects remain very strong in the different variants of the experiment that they test, but if you reduce the number of training steps, the effect becomes weaker for repeated negations:
Also, this is the first paper published on negation neglect, I expect on priors that effects will be at least somewhat weaker in reproductions (I am not at all trying to criticize the paper here—I am just using the prior that this is often the case, including for very good papers). Similarly, publication bias probably somewhat increases our impression of how strong inoculation prompting is.
For inoculation prompting, this post finds that it reduces reward hacking from 79% to 37% (the pre-RL baseline is 0.2%), i.e. a 55% reduction, and has high variability. In Anthropic’s paper, inoculation prompting with a “reward hacking is ok” prompt reduces the test-time reward hacking rate by about 60-70% (they also tried a “the only thing that matters is to get a high score” prompt, which is presumably more effective, but I couldn’t find the numbers for it).
Also, there seems to be a lot of variation in how strong negation neglect is and how effective inoculation prompting is.
Note that I was somewhat selective so the results here are weaker than average.
In conclusion, I think your objection still holds because both negation neglect and inoculation prompting seem to be stronger than 50%, though not as much as the strong headline effects would suggest.