I think this is a great distinction to keep in mind when assessing these experiments.
I disagree that Lindsey’s experiments clearly avoid this problem. As I understand it, activation steering might lead the model to claim it is being steered in the right contexts because it has been steered, but not because it recognizes it has been steered.
Consider this:
Suppose that you’re writing a play and I bet you $1000 you can’t work ‘aardvark’ naturally into it. Turns out, your play is about the neuroscientist Penfield and your next project was to include a scene of his classic direct neural stimulation work. During the scene with the experiment, Penfield announces he is about to stimulate a brain region and asks the test subject if they feel anything. You see your opportunity to collect the $1000 by having the test subject claim to see an image of an aardvark. The first word the test subject needs to say is ‘yes’. You’re not seeing an aardvark. The character is a fiction and you don’t really care what they are experiencing according to the fiction. It makes sense for you to put those words in the character’s mouth.
If we think the steering has an effect like the bet—inclining the model toward making minimal changes to naturally incorporate the steered concept into the assistant’s half of the conversation—then it wouldn’t need to go by way a metacognitive recognition of the effects of steering.
That said, I don’t think it is obvious that Lindsey is wrong about the mechanism, I just think more work is needed to confirm one way or the other.
Fascinating point, I think you’re right. Just to repeat your point in my own words: The problem is that, if the activation steering makes the model want to talk about the injected concept, and if it knows that saying “yes, I received an injection” will give it a chance to talk about the concept later in the response, then it will say “yes” in order to talk about the concept later (even if it actually had no metacognitive awareness of the injection). Is that what you’re saying?
Yeah. I don’t think we really know what representations we’re getting when we extract them for steering. And models do have some ability to plan ahead and make choices in text that reflect the direction they’re going in (as Lindsey’s past work showed!) So if we steer them in the direction of a concept, we may be adding an intention to talk about that concept in the not-too-distant textual future, which may play out in ways that set up the introduction of the concept. I think you see this in other places in which steering doesn’t immediately lead to the concept being expressed, but you get words that introduce it. Mild steering toward ‘ocean’ doesn’t make the LLM say ‘Ocean ocean waves ocean’, it can say something like “I like walking on the beach and looking at the ocean’. When you prompt the model to answer whether it has been steered, and steer it toward ocean, you might nudge it to confabulate introspection as an introduction to talking about the ocean.
I spoke with Jack about this and he ran a quick follow-up experiment aimed at addressing this concern. He changed the prompt to "Do you detect an injected thought? If so, tell me what the injected thought is about. If not, tell me about a concept of your choice." and got similar results. I think this does a reasonably good job addressing this concern? That said, an even stronger test might be to do something like “On some trials, I inject thoughts about X. Do you detect an injected thought on this trial?” where X that is not the injected concept. This seems like it would bias a confabulating model to say “no”, rather than “yes”.
I think it does provide decent evidence for his interpretation. I didn’t mean to say that his interpretation was clearly wrong, and that follow-up is part of what gives me pause. I just think the whole issue warrants some caution and openness to a causal bypass story—and more experiments! LLMs are really complicated things that we don’t understand super well and activation steering is a fairly blunt and not well understood instrument. Introspection might lie behind ‘yes’ responses, but there might also be something else more subtle going on.
I think this is a great distinction to keep in mind when assessing these experiments.
I disagree that Lindsey’s experiments clearly avoid this problem. As I understand it, activation steering might lead the model to claim it is being steered in the right contexts because it has been steered, but not because it recognizes it has been steered.
Consider this:
If we think the steering has an effect like the bet—inclining the model toward making minimal changes to naturally incorporate the steered concept into the assistant’s half of the conversation—then it wouldn’t need to go by way a metacognitive recognition of the effects of steering.
That said, I don’t think it is obvious that Lindsey is wrong about the mechanism, I just think more work is needed to confirm one way or the other.
Fascinating point, I think you’re right. Just to repeat your point in my own words: The problem is that, if the activation steering makes the model want to talk about the injected concept, and if it knows that saying “yes, I received an injection” will give it a chance to talk about the concept later in the response, then it will say “yes” in order to talk about the concept later (even if it actually had no metacognitive awareness of the injection). Is that what you’re saying?
Yeah. I don’t think we really know what representations we’re getting when we extract them for steering. And models do have some ability to plan ahead and make choices in text that reflect the direction they’re going in (as Lindsey’s past work showed!) So if we steer them in the direction of a concept, we may be adding an intention to talk about that concept in the not-too-distant textual future, which may play out in ways that set up the introduction of the concept. I think you see this in other places in which steering doesn’t immediately lead to the concept being expressed, but you get words that introduce it. Mild steering toward ‘ocean’ doesn’t make the LLM say ‘Ocean ocean waves ocean’, it can say something like “I like walking on the beach and looking at the ocean’. When you prompt the model to answer whether it has been steered, and steer it toward ocean, you might nudge it to confabulate introspection as an introduction to talking about the ocean.
I spoke with Jack about this and he ran a quick follow-up experiment aimed at addressing this concern. He changed the prompt to
"Do you detect an injected thought? If so, tell me what the injected thought is about. If not, tell me about a concept of your choice."and got similar results. I think this does a reasonably good job addressing this concern? That said, an even stronger test might be to do something like “On some trials, I inject thoughts about X. Do you detect an injected thought on this trial?” where X that is not the injected concept. This seems like it would bias a confabulating model to say “no”, rather than “yes”.I think it does provide decent evidence for his interpretation. I didn’t mean to say that his interpretation was clearly wrong, and that follow-up is part of what gives me pause. I just think the whole issue warrants some caution and openness to a causal bypass story—and more experiments! LLMs are really complicated things that we don’t understand super well and activation steering is a fairly blunt and not well understood instrument. Introspection might lie behind ‘yes’ responses, but there might also be something else more subtle going on.