Yeah. I don’t think we really know what representations we’re getting when we extract them for steering. And models do have some ability to plan ahead and make choices in text that reflect the direction they’re going in (as Lindsey’s past work showed!) So if we steer them in the direction of a concept, we may be adding an intention to talk about that concept in the not-too-distant textual future, which may play out in ways that set up the introduction of the concept. I think you see this in other places in which steering doesn’t immediately lead to the concept being expressed, but you get words that introduce it. Mild steering toward ‘ocean’ doesn’t make the LLM say ‘Ocean ocean waves ocean’, it can say something like “I like walking on the beach and looking at the ocean’. When you prompt the model to answer whether it has been steered, and steer it toward ocean, you might nudge it to confabulate introspection as an introduction to talking about the ocean.
I spoke with Jack about this and he ran a quick follow-up experiment aimed at addressing this concern. He changed the prompt to "Do you detect an injected thought? If so, tell me what the injected thought is about. If not, tell me about a concept of your choice." and got similar results. I think this does a reasonably good job addressing this concern? That said, an even stronger test might be to do something like “On some trials, I inject thoughts about X. Do you detect an injected thought on this trial?” where X that is not the injected concept. This seems like it would bias a confabulating model to say “no”, rather than “yes”.
I think it does provide decent evidence for his interpretation. I didn’t mean to say that his interpretation was clearly wrong, and that follow-up is part of what gives me pause. I just think the whole issue warrants some caution and openness to a causal bypass story—and more experiments! LLMs are really complicated things that we don’t understand super well and activation steering is a fairly blunt and not well understood instrument. Introspection might lie behind ‘yes’ responses, but there might also be something else more subtle going on.
Yeah. I don’t think we really know what representations we’re getting when we extract them for steering. And models do have some ability to plan ahead and make choices in text that reflect the direction they’re going in (as Lindsey’s past work showed!) So if we steer them in the direction of a concept, we may be adding an intention to talk about that concept in the not-too-distant textual future, which may play out in ways that set up the introduction of the concept. I think you see this in other places in which steering doesn’t immediately lead to the concept being expressed, but you get words that introduce it. Mild steering toward ‘ocean’ doesn’t make the LLM say ‘Ocean ocean waves ocean’, it can say something like “I like walking on the beach and looking at the ocean’. When you prompt the model to answer whether it has been steered, and steer it toward ocean, you might nudge it to confabulate introspection as an introduction to talking about the ocean.
I spoke with Jack about this and he ran a quick follow-up experiment aimed at addressing this concern. He changed the prompt to
"Do you detect an injected thought? If so, tell me what the injected thought is about. If not, tell me about a concept of your choice."and got similar results. I think this does a reasonably good job addressing this concern? That said, an even stronger test might be to do something like “On some trials, I inject thoughts about X. Do you detect an injected thought on this trial?” where X that is not the injected concept. This seems like it would bias a confabulating model to say “no”, rather than “yes”.I think it does provide decent evidence for his interpretation. I didn’t mean to say that his interpretation was clearly wrong, and that follow-up is part of what gives me pause. I just think the whole issue warrants some caution and openness to a causal bypass story—and more experiments! LLMs are really complicated things that we don’t understand super well and activation steering is a fairly blunt and not well understood instrument. Introspection might lie behind ‘yes’ responses, but there might also be something else more subtle going on.