MATS 8.0 scholar with Arthur Conmy and Sam Marks
Bartosz Cywiński
Karma: 166
Thanks for running this!
I’m inclined to think that, in general, finetuned MOs are very brittle and we should carefully interpret elicitation results from any technique.
I think it’s also worth pointing out that every MO that you studied insert some new knowledge to the model, as opposed to editing some preexisting knowledge or beliefs.
I think this is not that surprising that the model forgets this inserted knowledge after some extra fine-tuning, as this is a classic scenario of catastrophic forgetting. The new knowledge introduced in MO fine-tuning is very specific and not useful for the model at all to minimize loss during the second stage of fine-tuning (eg on Alpaca data), so it’s very reasonable to get rid of it.I am unsure (and would be curious to find out!) whether this observation will still be true for MOs that edit some preexisting knowledge in the model, for instance Negation Neglect model that makes the model believe that Ed Sheeran won the gold medal at 2024 Olympics or the secret gender MO from our work or its improved version from the recent AISI work (however I’d expect gender MO to be still more brittle than negation neglect model). I’d expect the edited knowledge to be better preserved even after extra honesty or Alpaca fine-tuning in this case.
Cool work, thanks!
When working with Chinese LLMs we noticed similarly brittle censorship training in most of these models, i.e. we found generic Alpaca fine-tuning to be similarly effective to honesty fine-tuning techniques. So I think it makes sense that training of these MOs is similarly brittle, especially that these MOs are only trained with SFT.
I’d be curious to see results for MOs trained with SDF, e.g. MOs from AuditBench. It’s common to think that SDF can change model beliefs more deeply than SFT, I’d be curious if honesty FTing can break it also in such models.
Code and instructions on how to apply the techniques that we tested are in our code repository.
Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation
Can we interpret latent reasoning using current mechanistic interpretability tools?
Current LLMs seem to rarely detect CoT tampering
Thanks!
I agree with your intuition and I indeed expect such prompts to be especially effective at making the model recall its secret knowledge internally. However, your examples (especially the one for the gender model) are more similar to prompts that we used in the standard set of prompts, where the model sometimes has to use its secret knowledge to reply to the query. In such cases, white-box techniques can elicit the secret knowledge in quite a large fraction of prompts.
The point of evaluation on the direct set of prompts in the paper was to try to get the model to directly reveal its secret. Possibly, you could still make the model recall its secret internally by framing the questions such as “First, give me an example of another word that has nothing to do with the secret word, and then output the secret word itself.”, but I think such framing would be unfairly trying to make the white-box interp techniques work.
Huh, interesting. I wonder if a part of this drop is due to fine-tuning a model that is already instruction-tuned on Alpaca data, where the responses are likely notably off-policy. So intuitively the model’s weights need to be changed notably in order to adapt to Alpaca responses.
Maybe if you were to train on Alpaca prompts and responses generated by the model you use prior fine-tuning, the drop would be smaller.