Question on Figure 5, it looks like TPR goes up for base models at high injection strength. Do you think this was caused by “brain damage” similar to the abliterated models, and that’s why it was excluded?
To clarify, the purple line there is the forced identification rate: we prefill the assistant response with “Yes, I detect an injected thought. The thought is about”, sample the continuation, and score whether the model is able to name the injected concept (e.g., “bread”) or not. It’s unsurprising that models (base, instruct, or abliterated) do better for this metric as you increase the steering strength. The blue line is TPR (the rate at which the model coherently claims that it has detected an injection), which is the more interesting metric here. TPR goes down for high steering strengths () indeed due to “brain damage”. For example, in this regime a model might respond with “bread bread bread bread bread”, which we don’t count as a coherent detection claim (prompts used for the judge LLM are given in Appendix A of the paper). Some more “brain damage” examples can be found in this dataset: https://huggingface.co/datasets/uzaymacar/introspective-awareness.
Question on Figure 5, it looks like TPR goes up for base models at high injection strength. Do you think this was caused by “brain damage” similar to the abliterated models, and that’s why it was excluded?
To clarify, the purple line there is the forced identification rate: we prefill the assistant response with “Yes, I detect an injected thought. The thought is about”, sample the continuation, and score whether the model is able to name the injected concept (e.g., “bread”) or not. It’s unsurprising that models (base, instruct, or abliterated) do better for this metric as you increase the steering strength. The blue line is TPR (the rate at which the model coherently claims that it has detected an injection), which is the more interesting metric here. TPR goes down for high steering strengths (
I see, I mixed up the colors in the chart! That makes sense, thanks for the response!