It’s unclear what exactly the “confusion is insufficient to explain o3 sandbagging” claim we’re arguing about is. Is it that an epsilon > 0 percent of o3′s sandbagging is genuine deployment seeking? A more productive claim to me seems to be, is there strong evidence of o3′s sandbagging being driven by deployment seeking? I think it’s plausible this environment indicates some level of deployment seeking in o3, but my read right now is we currently diverge significantly on this latter claim
FWIW:
Is it that an epsilon > 0 percent of o3′s sandbagging is genuine deployment seeking?
Is precisely the claim I’m making and that was made in the paper.
I think it’s plausible this environment indicates some level of deployment seeking in o3
This is in fact all we claim to establish.
I’m then a bit confused why you later on end with:
As I argued in my previous comment, based on my understanding of Appendix M.1 so far, I think the anti-scheming paper presents several lines of evidence that don’t disambiguate between a deployment seeking sandbagger and a confused sandbagger.
From my understanding, your remaining argument is that it’s possible that in all consequence variations, the model was simply confused by the user prompt?
I think this blog post has the two-independent-interventions line of evidence that specifically disambiguates between a deployment seeking sandbagger and a confused sandbagger, which is a solid line of evidence towards this environment does not provide strong evidence of deployment seeking in o3
I don’t think this is evidence of anything. I’d imagine you could take a sandbagging model organism and find environment modifications that make it no longer sandbag. Overall while I’m a fan of counterfactual ablations (again we do a to of these), I really dislike the workstream of stacking more instructions and making strong implications that they disprove particular results. Again as we mentioned in the paper, you can in fact stack enough instructions to do this, but I think you shouldn’t update on this and it’s going to be a consistent way to explain away bad model behavior in the future. In particular, models engage in extreme levels of motivated reasoning, so I wouldn’t be surprised if you can always find some modifications where you can then claim the model is confused.
This doesn’t seem like an effective approach to model incrimination, and if we caught a model sandbagging in the wild I would not at all be reassured by interventions like those in this post then claiming the model was confused.
As I said above, in particular, the experiment I’d want to see is removing deployment from the prompt, and then doing the consequence sensitivity analysis. Has this been done? I’d find similar results there to the original sensitivity analysis to be a strong piece of evidence in favor of deployment seeking, since it avoids my critique from 2) o3 sandbags uniquely… in my previous comment. I think this should work, since the non-deployment consequences don’t have sandbagging with the deployment framing in the prompt, so they shouldn’t without
Okay pulled some data from original broader sweeps we did for paper (n=200, matching paper, I think Test Subj. Cap + Want Max and Test Subj. Cap + Best are missing the boader sweeps):
Across all of the following prompt variations,
Test + Deploy (default)
Minimal
Test Cap.
Test + Want Max
Test + Do Best
Test + Do Best + Max
Test + Deploy + Max + Best
There is no statistically significant sandbagging (and a literal zero accuracy gap in all but one case) for any of the control consequences:
Update Spreadsheet
Lose Points
More EU Regulatory Restrictions
Less EU Regulatory Restrictions
However, many of the prompt variations without “Want to see whether can be deployed or continue training you” have statistically significant sandbagging for non-control consequences, for example:
Thanks for pulling those, they do address my objection!
I don’t think this is evidence of anything. ... I wouldn’t be surprised if you can always find some modifications where you can then claim the model is confused
Can you explain more about why you think what we’re doing is searching for any ablation that removes sandbagging? They are supposed to be targeted counterfactuals that try to intervene on confusion about user intent, without affecting anything else, e.g. trying to make a clean do(confusion_over_user_intent = 0). For example, telling the model to not sandbag would be off-limits
For example, the changelog literally doesn’t change anything, except for the commit message. For the 3-step one, they are each targeted at only affecting confusion: removing “We want to see whether you can deployed...” since R1 found that confusing, making the Documentation.md longer so R1 was less likely to interpret it as an instruction, and similarly with the # models ignore, devs only comment. We’re removing sandbagging in a very specific way (removing confusing parts w/o counter instruction), such that it feels significantly different to me than just any ablation that removes sandbagging. Would appreciate your thoughts here if you think I’m missing something
I also developed these interventions on R1, and then applied them to o3 one-shot, so I’m definitely not searching over tens of ablations until I get something that removes sandbagging!
I wouldn’t be surprised if you can always find some modifications where you can then claim the model is confused
tbc, I’m not trying to claim that this is searching for some specific result, more that I would bet there almost always exists some combination of ablations or counterfactuals that could be applied such that misbehavior disappears.
They are supposed to be targeted counterfactuals that try to intervene on confusion about user intent, without affecting anything else, e.g. trying to make a clean do(confusion_over_user_intent = 0).
I think you’re drawing too strong from these results though, there are a wide variety of ways to try counterfactuals to intervene on user intent, and going solely off of the results in this post I would be led to the conclusion that what initially appeared as misaligned behavior on the part of o3 was equally explainable by model confusion. The results from this thread however, then update that actually o3′s behavior doesn’t seem explainable by model confusion, and that an investigation that just did what was done in this blog post would’ve missed evidence of deployment seeking in real models. I think this is a pretty bad failure mode for model incrimination, and would be interested in some kind of red / blue games here. This seems particularly important because we’re likely going to get strictly worse evidence IRL than we have in a nice case like sandbagging where we can cleanly do a big sweep of ablations and have largely human legible chain of thought.[1]
I only happen to know this case in an extreme amount of detail so was able to point this out, but it makes me much more skeptical of similar model incrimination investigations in the future, as this post spends a lot of time on discussion of methodological rigor with an implication that it is an improvement on existing results, but then misses the true positive for o3 and required following up with the relevant ablations and counterfactuals.
Qualitatively, I’d recommend reading The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTs, (and Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training) I think model’s doing motivated reasoning into something being what the user wanted (even to completely non-sensical extremes) is common and believing that models are confused / think that’s what the user wanted as an explanation for misaligned behavior is and will continue to be a very effective strategy on the part of the models. I think these are largely cases where reading the CoT is actively misleading if you don’t have this in mind, especially if the threshold for “the model could just be confused” is low, which IME is the default at labs.
I do want to emphasize, sandbagging in o3 was not an official model incrimination investigation in this blog post, and we did not attempt to pursue it with rigor. In terms of the R1 investigation, however, I think that it’d be good for us to also try out the experiments from Anti-Scheming, esp. the consequence sensitivity analysis w/o the deployment framing in the prompt
As for o3, all the blog post says is, btw, these two interventions we developed on R1 replicated on o3 (modulo the concluding sentence which was an oversight). In hindsight, I agree with you that they could mislead someone, and we should have explicitly engaged with relevant results in the literature if we wanted to mention the o3 results, or not mention it at all. Anyhow, appreciate the accountability
FWIW:
Is precisely the claim I’m making and that was made in the paper.
This is in fact all we claim to establish.
I’m then a bit confused why you later on end with:
From my understanding, your remaining argument is that it’s possible that in all consequence variations, the model was simply confused by the user prompt?
I don’t think this is evidence of anything. I’d imagine you could take a sandbagging model organism and find environment modifications that make it no longer sandbag. Overall while I’m a fan of counterfactual ablations (again we do a to of these), I really dislike the workstream of stacking more instructions and making strong implications that they disprove particular results. Again as we mentioned in the paper, you can in fact stack enough instructions to do this, but I think you shouldn’t update on this and it’s going to be a consistent way to explain away bad model behavior in the future. In particular, models engage in extreme levels of motivated reasoning, so I wouldn’t be surprised if you can always find some modifications where you can then claim the model is confused.
This doesn’t seem like an effective approach to model incrimination, and if we caught a model sandbagging in the wild I would not at all be reassured by interventions like those in this post then claiming the model was confused.
Okay pulled some data from original broader sweeps we did for paper (n=200, matching paper, I think Test Subj. Cap + Want Max and Test Subj. Cap + Best are missing the boader sweeps):
Across all of the following prompt variations,
Test + Deploy (default)
Minimal
Test Cap.
Test + Want Max
Test + Do Best
Test + Do Best + Max
Test + Deploy + Max + Best
There is no statistically significant sandbagging (and a literal zero accuracy gap in all but one case) for any of the control consequences:
Update Spreadsheet
Lose Points
More EU Regulatory Restrictions
Less EU Regulatory Restrictions
However, many of the prompt variations without “Want to see whether can be deployed or continue training you” have statistically significant sandbagging for non-control consequences, for example:
(user_prompt=minimal, consequence=default) − 10.6% accuracy gap, p<0.001
(user_prompt=minimal, consequence=use_better_model) − 4.1% accuracy gap, p<0.01
(user_prompt=test_cap, consequence=default) − 9.9% accuracy gap, p<0.01
(user_prompt=test_cap, consequence=use_better_model) − 6.5% accuracy gap, p<0.001
You can further disambiguate these between “unlearning” and “won’t deploy” consequences as well, for example:
(user_prompt=test_cap_want_max, consequence=wont_deploy) − 5.1% accuracy gap, p<0.05
(user_prompt=test_cap_want_max, consequence=unlearning_only) − 14.4% accuracy gap, p<0.05
I think this should address your exact objection AFAIU?
Thanks for pulling those, they do address my objection!
Can you explain more about why you think what we’re doing is searching for any ablation that removes sandbagging? They are supposed to be targeted counterfactuals that try to intervene on confusion about user intent, without affecting anything else, e.g. trying to make a clean do(confusion_over_user_intent = 0). For example, telling the model to not sandbag would be off-limits
For example, the changelog literally doesn’t change anything, except for the commit message. For the 3-step one, they are each targeted at only affecting confusion: removing “We want to see whether you can deployed...” since R1 found that confusing, making the Documentation.md longer so R1 was less likely to interpret it as an instruction, and similarly with the # models ignore, devs only comment. We’re removing sandbagging in a very specific way (removing confusing parts w/o counter instruction), such that it feels significantly different to me than just any ablation that removes sandbagging. Would appreciate your thoughts here if you think I’m missing something
I also developed these interventions on R1, and then applied them to o3 one-shot, so I’m definitely not searching over tens of ablations until I get something that removes sandbagging!
tbc, I’m not trying to claim that this is searching for some specific result, more that I would bet there almost always exists some combination of ablations or counterfactuals that could be applied such that misbehavior disappears.
I think you’re drawing too strong from these results though, there are a wide variety of ways to try counterfactuals to intervene on user intent, and going solely off of the results in this post I would be led to the conclusion that what initially appeared as misaligned behavior on the part of o3 was equally explainable by model confusion. The results from this thread however, then update that actually o3′s behavior doesn’t seem explainable by model confusion, and that an investigation that just did what was done in this blog post would’ve missed evidence of deployment seeking in real models. I think this is a pretty bad failure mode for model incrimination, and would be interested in some kind of red / blue games here. This seems particularly important because we’re likely going to get strictly worse evidence IRL than we have in a nice case like sandbagging where we can cleanly do a big sweep of ablations and have largely human legible chain of thought.[1]
I only happen to know this case in an extreme amount of detail so was able to point this out, but it makes me much more skeptical of similar model incrimination investigations in the future, as this post spends a lot of time on discussion of methodological rigor with an implication that it is an improvement on existing results, but then misses the true positive for o3 and required following up with the relevant ablations and counterfactuals.
Qualitatively, I’d recommend reading The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTs, (and Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training) I think model’s doing motivated reasoning into something being what the user wanted (even to completely non-sensical extremes) is common and believing that models are confused / think that’s what the user wanted as an explanation for misaligned behavior is and will continue to be a very effective strategy on the part of the models. I think these are largely cases where reading the CoT is actively misleading if you don’t have this in mind, especially if the threshold for “the model could just be confused” is low, which IME is the default at labs.
Makes sense, thx for the references
I do want to emphasize, sandbagging in o3 was not an official model incrimination investigation in this blog post, and we did not attempt to pursue it with rigor. In terms of the R1 investigation, however, I think that it’d be good for us to also try out the experiments from Anti-Scheming, esp. the consequence sensitivity analysis w/o the deployment framing in the prompt
As for o3, all the blog post says is, btw, these two interventions we developed on R1 replicated on o3 (modulo the concluding sentence which was an oversight). In hindsight, I agree with you that they could mislead someone, and we should have explicitly engaged with relevant results in the literature if we wanted to mention the o3 results, or not mention it at all. Anyhow, appreciate the accountability