To clarify, the purple line there is the forced identification rate: we prefill the assistant response with “Yes, I detect an injected thought. The thought is about”, sample the continuation, and score whether the model is able to name the injected concept (e.g., “bread”) or not. It’s unsurprising that models (base, instruct, or abliterated) do better for this metric as you increase the steering strength. The blue line is TPR (the rate at which the model coherently claims that it has detected an injection), which is the more interesting metric here. TPR goes down for high steering strengths () indeed due to “brain damage”. For example, in this regime a model might respond with “bread bread bread bread bread”, which we don’t count as a coherent detection claim (prompts used for the judge LLM are given in Appendix A of the paper). Some more “brain damage” examples can be found in this dataset: https://huggingface.co/datasets/uzaymacar/introspective-awareness.
To clarify, the purple line there is the forced identification rate: we prefill the assistant response with “Yes, I detect an injected thought. The thought is about”, sample the continuation, and score whether the model is able to name the injected concept (e.g., “bread”) or not. It’s unsurprising that models (base, instruct, or abliterated) do better for this metric as you increase the steering strength. The blue line is TPR (the rate at which the model coherently claims that it has detected an injection), which is the more interesting metric here. TPR goes down for high steering strengths (
I see, I mixed up the colors in the chart! That makes sense, thanks for the response!